Papers with fine-tuning scenarios
CoLLiE: Collaborative Training of Large Language Models in an Efficient Way (2023.emnlp-demo)
Copied to clipboard
Kai Lv, Shuo Zhang, Tianle Gu, Shuhao Xing, Jiawei Hong, Keyu Chen, Xiaoran Liu, Yuqing Yang, Honglin Guo, Tengxiao Liu, Yu Sun, Qipeng Guo, Hang Yan, Xipeng Qiu
| Challenge: | Large language models (LLMs) are increasingly pivotal in a wide range of tasks . however, the resources required for training these models necessitate efficient solutions . |
| Approach: | They propose a library that facilitates collaborative training of large language models . they use 3D parallelism, parameter-efficient fine-tuning methods and optimizers . |
| Outcome: | The proposed library has proven superior training efficiency in comparison with prevalent solutions in pre-training and fine-tuning scenarios. |
CLEAN–EVAL: Clean Evaluation on Contaminated Large Language Models (2024.findings-naacl)
Copied to clipboard
Wenhong Zhu, Hongkun Hao, Zhiwei He, Yun-Ze Song, Jiao Yueyang, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, Hongyuan Lu
| Challenge: | Existing methods to evaluate large language models are prone to data contamination. |
| Approach: | They propose a method which parses contaminated data and back-translates it into a candidate set. |
| Outcome: | The proposed method reduces data contamination and evaluates the LLMs more cleanly. |
Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing approaches for fine-tuning large language models require a trade-off between exact gradients with high memory and low memory with noisy estimates (MeZO). |
| Approach: | They propose a method which derivates gradients from LoRA's low-rank structure and manually deriving backward passes to exploit the low-level structure. |
| Outcome: | The proposed method reduces peak memory from 361MB to 136MB for Qwen2.5-0.5B, enabling fine-tuning scenarios previously infeasible on memory-constrained devices. |
FroM: Frobenius Norm-Based Data-Free Adaptive Model Merging (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new adaptive merging method is proposed to improve fine-tuning performance . traditional methods often encounter task interference when merging full fine-uning models . |
| Approach: | They propose an adaptive merging method that directly measures model parameters using the Frobenius norm . |
| Outcome: | The proposed method outperforms baseline methods in various fine-tuning scenarios. |
Investigating the Zone of Proximal Development of Language Models for In-Context Learning (2025.findings-naacl)
Copied to clipboard
| Challenge: | In-context learning is a dynamic and progressive process where learners integrate new information into their knowledge base through interactions with the environment. |
| Approach: | They propose a learning analytics framework to analyze the in-context learning behavior of large language models (LLMs) through the lens of the Zone of Proximal Development (ZPD), an established theory in educational psychology. |
| Outcome: | The proposed framework improves inference and fine-tuning scenarios by selectively applying it to queries that are most likely to benefit from demonstrations. |
KS-Lottery: Finding Certified Lottery Tickets for Multilingual Transfer in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that a small subset of parameters is highly effective in fine-tuning . prior work shows that there are a few additional parameters corresponding to an intrinsic dimension in a well-trained Large Language Model. |
| Approach: | They propose a method to identify a small subset of LLM parameters highly effective in multilingual fine-tuning. |
| Outcome: | The proposed method can find the certified winning tickets in the embedding layer, and fine-tuning on the found parameters is guaranteed to perform as well as full fine- tuning. |